"When to Stop" Waterloo (Cormack) Participation in the TREC 2016 Total Recall Track

In the course of developing tools for the 2015 Total Recal Track, Track Co-Coordinators Gordon V. Cormack and Maura R. Grossman created an autonomous continuous active learning (“CAL”) system, which was provided to participants as the baseline model implementation (“BMI”) [http://plg.uwaterloo.ca/∼gvcormac/trecvm/]. BMI employs the technology-assisted review (“TAR”) approach described by Cormack and Grossman [2]; the only difference is that BMI employs logistic regression implemented by Sofia ML [https://code.google.com/p/sofia-ml/], instead of SVMlight [http://svmlight.joachims.org/]. BMI was reprised, unchanged from TREC 2015, except for the addition of a default “call-your-shot” stopping rule indicating the system’s estimate of the point at which a reasonable compromise between recall and effort had been achieved. The Waterloo (Cormack) team submitted runs using BMI for the “Athome” and “Sandbox” tasks. The only change that was made to BMI was to incorporate two different “call-your-shot” criteria that the authors had previously reported at SIGIR 2016 [1]: